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Abstract 

To each partition A = (Ai,A2,...) with distinct parts we assign the probability 
Q\{x)P\(y) / Z where Q\ and P\ are the Schur Q-functions and Z is a normalization 
constant. This measure, which we call the shifted Schur measure, is analogous to the 
much-studied Schur measure. For the specialization of the first m coordinates of x 
and the first n coordinates of y equal to a (0 < a < 1) and the rest equal to zero, we 
derive a limit law for Ai as m, n — > oo with r = m/n fixed. For the Schur measure the 
a-specialization limit law was derived by Johansson. Our main result implies that the 
two limit laws are identical. 
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1 Introduction 

The Schur measure [Okl] assigns to each partition A = (Ai, A2, • • •) the weight 

s\(x) s x (y) 

where s\ are the Schur functions. (See, e.g., [Mac, St].) Thus 

J>a(x)sa(2/) (LI) 

xev 

\ 1 <h 

is the (unnormalized) probability that Ai, the number of boxes in the first row of the asso- 
ciated Young diagram, is less than or equal to h. Here V denotes the set of all partitions. 
The normalization constant, Z, is determined from the h — > 00 limit 

Z:=^ SA (x) SA ( 2/ ) = n T 3^-' (I- 2 ) 

where the last equality is the Cauchy identity for Schur functions. A theorem of Gessel [Ge] 
expresses the partition sum (1.1) as an h x h Toeplitz determinant Dh(ip). It follows from 
this that the normalization constant is also given by 

Z = lim D h (ip). 

h— >oo 
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This limit can be explicitly computed by an application of the strong Szego limit theorem. 
(See, e.g., [BS]) and thereby the Cauchy identity reappears. 

The Toeplitz determinant, or the Fredholm determinant coming from the Borodin-Okounkov 
identity [BO, BW], is the starting point in the analysis of limit laws for Ai. This analysis to- 
gether with the Robinson-Schensted-Knuth (RSK) correspondence gives a new class of limit 
laws, first discovered in the context of random matrix theory [TW1, TW2], for a number of 
probability models. Indeed, the result of Baik, Deift and Johansson [BDJ] for the limit law 
of the length, £n{tt), of the longest increasing subsequence in a random permutation ir G Sn 
is the now classic example: Exponential specialization of the Gessel identity together with 
the RSK correspondence 1 shows that 

oo N 
N=0 

is an hx h Toeplitz determinant with symbol tp(z) = ev^O+i/^ ^ n aS ymptotic analysis of this 
Toeplitz determinant (using the steepest descent method for Riemann-Hilbert problems [DZ]) 
shows that 

^"pw 1 <•)=*« 

where F 2 is the limiting distribution of the largest eigenvalue (suitably centered and normal- 
ized) in the Gaussian Unitary Ensemble [TW1]. Similar results hold for longest increasing 
subsequences in symmetrized random permutations [BR1, BR2] and random words [AvM, 
Jo2, ITW, TW3], for height fluctuations in various growth models [BR3, GTW1, GTW2, 
Jol], and tiling problems [Jo3], as well as extensions to the other rows of the Young dia- 
gram [BOO, Jo2, Ok2]. 

In the theory of symmetric functions there are many important generalizations of Schur 
functions [Mac]. These generalizations satisfy Cauchy identities and it is natural to inquire 
about more general Gessel identities. However, one quickly sees that without determinantal 
formulas of the type that exist for Schur functions (the Jacobi-Trudi identity), Gessel iden- 
tities seem unlikely. Nevertheless, the question of possible limit laws for sums of the type 
(1.1) remains interesting. 

This paper initiates work in this direction. Instead of Schur functions we shall work with 
Schur Q-functions which have pfaffian representations. These functions, introduced by Schur 
in 1911 in his analysis of the projective representation of the symmetric group, now have a 
combinatorial theory that parallels the combinatorial theory of Schur functions. This theory, 
due to Sagan [Sa] and Worley [Wo] (see also [Ste, HH]), is based on a shifted version of the 
RSK algorithm. Whereas the usual RSK algorithm associates bijectively to each N-matrix 
A a pair of semistandard Young tableaux, the shifted RSK algorithm associates bijectively 
to each P-matrix 2 A a pair of shifted Young tableaux. There is a notion of increasing paths 
and the length of the maximal path, L(A), equals the number of boxes in the first row of 

J RSK associates bijectively to each permutation ir a pair of standard Young tableaux (P, Q) of the same 
shape A such that £n(tt) = Ai. See, e.g., [St]. 

2 Informally, a P-matrix is an N- matrix where we allow the nonzero entries to be either marked or un- 
marked. Precise definitions arc given below. 
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the shifted tableau. Thus it is natural to assign to each partition A into distinct parts, i.e., 
a strict partition, the probability 

P({X}) = ^Qx(x)P x (y) (1.3) 

where Q\ and P\ are the Schur Q-functions and Z is a normalization constant. We call this 
measure the shifted Schur measure. 

At first our analysis is for general parameters x and y appearing in the shifted Schur 
measure, and we find that there is indeed a Gessel identity. (It follows from the Ishikawa- 
Wakayama pfaffian summation formula [IW].) Then we specialize the measure by choosing 
the first m coordinates of x and the first n coordinates of y equal to a (0 < a < 1) and the 
rest equal to zero. We call this a-specialization and denote the resulting specialized shifted 
Schur measure by P CT where a = (m, n, a) denotes the parameters of the measure. Now, 
however, the matrix on the right side of the Gessel identity is not Toeplitz and so the earlier 
analytical methods are not immediately available to us. Nevertheless, we do find that the 
distribution function for L(A) = Ai can be expressed in terms of the Fredholm determinant 
of an operator which is a perturbation of a direct sum of products of Hankel operators. In 
the end we can show that the trace norm of the perturbations tend to zero and are able to 
determine the asymptotics. 

We asume 3 that m/n = r is a constant satisfying a 2 < r < a~ 2 . Our main result is 

Theorem Let P a denote the a- specialized shifted Schur measure with r satisfying the stated 
restriction. Then there exist constants c\ = ci(a,r) and c 2 = c 2 (a,r) such that 

^ ( Ai — c\ n \ , . 
lim P CT 1 < s )= F 2 (s). 

For r = 1 the constants have a particularly simple form; namely, 

( ^ 4a , 1N (2a(l + 6a 2 + a 4 )) 1/3 

ci(a,l) = - 2 and c 2 {a, 1) = . 

1 — cr 1 — cr 

Expressions for c\ and c 2 in general are given in §6. For the Schur measure the corresponding 
a-specialization limit law was derived by Johansson [Jol]; as the theorem shows, the two 
limit laws are identical. 4 The table of contents provides a description of the layout of this 
paper. 

2 Schur Q-Functions 

This section and the next summarize the properties of the Schur Q-functions and the shifted 
RSK algorithm that we will need in this paper. The material is not new on our part. It 

3 The stated restriction on r is very likely unnecessary for the validity of the final result. Some details of 
the proof would be different in the other cases but we did not carry them out. 

4 We note that our a is related to Johansson's q by q = a 2 . For the a-specialized Schur measure, 
ci(a, 1) = 2a/(l - a) and c 2 (a, 1) = a 1 /^! + a) 1 / 3 /(l - a). 
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is presented to establish the notation used in subsequent sections as well as a convenience 
to the reader. A complete presentations can be found in the books by Macdonald [Mac], 
Hoffman and Humphreys [HH], the papers by Sagan [Sa], Stembridge [Ste], and the thesis 
of Worley [Wo] . 

If A = (Ai, A2, . . .) is a partition of n, we denote this by A h n. The length of A is denoted 
by ^(A). Let V n denote the set of all partitions of n and V := \J^ =Q V n - (Po is the empty 
partition.) Introduce T> n C V n : the set of partitions of n into distinct parts. For example 

D 6 = {{6}, {5,1}, {4, 2}, {3, 2,1}}. 

Let V := U^Lo^™> ^ ne se ^ °^ an partitions into distinct parts. If A h n is a partition with 
distinct parts, we denote this by A |= n and call A a strict partition of n. 

Associated to a strict partition A is a shifted shape S(X). One starts with the usual Young 
diagram Y(X) and for i = 1,2,..., £(X) simply indents the i^ 1 row to the right by i — 1 boxes. 
The result is S(X). We usually use A G T> both to denote a strict partition and the shifted 
shape S'(A). 

We let N denote the set of positve integers and 

P={1 / ,1,2',2,3',3,...} 

with the ordering 

1' < 1 < 2' < 2 < 3' < 3 < • • • . 

We call the elements either marked or unmarked depending on whether the element is primed 
or not. When we do not wish to distinguish a marked element m! from the unmarked element 
m, we write m*. A (generalized) shifted Young tableau, T, is an assignment of elements of P 
to a shifted shape A having the properties 

Tl T is weakly increasing across rows and down columns. 

T2 For each integer m*, there is at most one m' in each row and at most one m in each 
column of T. (Thus the marked elements are strictly increasing across rows of T and 
the unmarked elements are strictly increasing down columns of T.) 

An example of a shifted tableau of shape (7, 5, 3, 2, 1) is 



1 2' 


2 


2 


5' 


2' 2 


3' 


4 


5 


3' 


4 


5 






6 


r 








7. 





To each shifted tableau T we associate a monomial 

X — x 2 A m 

where a m is the number of times m* appears in T. Thus as far as the monomial is concerned, 
we do not distinguish between marked and unmarked elements. In the above example, 

T 2522322 
ju — 1 2 3 4 5 6 7* 
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Let A be a strict partition of n. The Schur Q-function, the generating function of shifted 
tableaux of shape A, is 

Q A (x):=J> T , (2.1) 

T 

where the sum runs over all shifted tableaux of shape X \= n. The Schur Q-function is the 
analogue of the Schur function s\ when one replaces semistandard Young tableaux of shape 
A by shifted tableaux of shape A. (Of course, here A must be a strict partition.) It will be 
convenient to introduce the Schur P-function 

We remark that a shifted tableau T of shifted shape A |= n is called standard if it has no 
marked elements and uses each unmarked letter 1,2, ... ,n exactly once. Schur showed that 
the number of standard shifted tableaux of shape A |= n, A = (Ai, A2, . . . , A^) is 

/s " = A 1 !A 2 !---A,! i n A~Ta/ ^ 
This should be compared with the number of standard Young tableaux 



f x PJPJ...P,] n & 



l<i<j<k 

where A = (Ai, A2, • • • , Xk) and £j = Xj + k—j. The number of semistandard Young tableaux 
of shape A that can be formed using the integers 1, 2, . . . , n is 

n 

d x (n) = s A (l~^~r,0,0,...). 

Similarly, the number of shifted tableaux of shape A that can be formed using the integers 
l*,2*,...,n* is 

n 

^H = g A (C^~T,o,o,...). 

This specialization of Q\ will be important below. 
The Schur Q-functions satisfy a Cauchy identity 



00 



\eT> i,j=l 



'EQx(x)Px<v)= l[\±^- = Z. (2.3) 



The right-hand side enumerates all matrices A whose entries are chosen from P U {0}: the 
denominator counts matrices with entries in N U {0}, while the numerator accounts for the 
primes. We call these matrices P-matrices. The above product will frequently be specialized 
to x — (xi, X2, ■ ■ ■ , x m , 0, . . .) and y = (yi, y%, ■ ■ ■ , y n , 0, . . .). We use the same symbol Z to 
denote this specialization. It will be clear from the context how to interpret Z. 
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Define symmetric functions by 

i=l % k=0 

(When necessary to indicate the dependence upon x, we write Q(t,x).) It follows from 
Q(t)Q(-t) = 1 that 

m—l 1 

m2 



r=l 

which shows that q 2m , G Q[<Zi, Q2, ■ ■ ■ , fcm-i] an d hence by induction on m, 

<?2m e Q[<?1,<?3,95, • • • ,<?2m-l]- 

Denote by T the subring of A generated by the q r : 

r = Z[gi,g 3 , . . .]. 

If A = (Ai, A 2 , . . .) we let 

q\ :=gAi?A a -". 

It is known that the g> with A strict form a Z-basis of I\ 

We now give the classical definition of the Schur Q-function. (Of course, in this presen- 
tation it is a theorem.) If A is a strict partition of length < n, then Q\ equals the coefficient 
of t x := t^t* 2 ■ ■ ■ in 

n 

Q(h, t 2 ,...,t n )=Y[ Q(U) J] F{t~Hj) 

i=l i<j 

where 

^) = ^ = 1 + 2E(-!)V 

y r>l 

and Q is defined by (2.4); in particular, for r > s 

Q(r,s) = (J^J y /" t^-H^Fih/t^Q^Qih) dh dt 2 (2.5) 

where the contours could be chosen to be circles with \t 2 \ < \t±\. 5 
Here are some additional properties of Schur Q-functions: 

1. The Q\, A strict, form a Z-basis of T. 



5 This requires that 1 £ f 1 and that the poles x~ x lie outside the contours. Notice that if the t\ contour 
were deformed to one inside the ti contour then since Q(t)Q(—t) = 1 the residue at the pole t\ = — 1 2 crossed 
would be 2t^ r_s_ . The integral of this equals zero as long as r and s are not both zero. This shows that 
the contours can also be chosen so that \t2\ > \t\\. Equivalcntly, the integral representation holds for r < s 
as well. 
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2. Using (2.5) and the expansion for F we have for r > s > 

s 

Q(r,s) = Qr-qs + 2^2(-l) l q r+i q s _i (2.6) 

i=l 

For r < s we define Qr r s ) = — Q(s,r)- Now let A be a strict partition which we write 
in the form A = (Ai, A2, . . . , \2n) where Ai > A2 > ■ ■ • > \2n > 0. Define the 2n x 2n 
antisymmetric matrix 

M x = {Q M )) , 

then we have 

Q A = pf(M A ) (2.7) 

where pf denotes the pfafnan. 



3 Shifted RSK Algorithm 

For later convenience, we use a nonstandard labeling of the matrix A: rows are numbered 
starting at the lower left-hand corner of A and columns have the usual left-to-right labeling. 
To each IP-matrix A we (bijectively) associate a biword wa as follows. For a fixed column 
index we scan the matrix for increasing values of the row index. If the (i, j) entry is unmarked 

with value we repeat the pair ( ^, ^ times in wa- If the element is marked, the 
i of the first pair ( ^ J appearing in wa is marked. For example, if 




then 



1 1 1 1 1 2 2 3 3 3 
V 1 1 2' 3 3 3 1 2' 2 



A description of the shifted RSK algorithm is more involved than the usual RSK algo- 
rithm, though the general features remain the same. Namely, there is a row bumping (and 
column bumping) procedure which when iterated on a sequence a whose elements are in P 
gives a shifted tableaux S, the insertion tableaux. (This is applied to the sequence in the 
bottom half of the biword wa-) The top half of wa gives a recording tableaux T. We now 
state the final result, referring the interested reader to either [HH] or [Sa]. 

Theorem (Sagan, Worley). There is a bijective correspondence betweenF -matrices A = (0^-) 
and ordered pairs (S, T) of shifted tableaux of the same shape, such that T has no marked 
letters on its main diagonal. The correspondence has the property that £\ a*- is the number 
of entries t of T for which t* = j , whereas a*j is the number of entries s of S for which 
s* = i. (Recall that the * means we do not distinguish between a marked or unmarked form 
of an integer.) 
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We call the matrix S, respectively T, to be of type s, respectively t. 

An important property of the RSK algorithm is its relationship to increasing subsequences 
of maximal length in the biword wa (equivalently, increasing paths in the matrix A of 
maximal weight). The shifted RSK algorithm of Sagan and Worley shares a similar property 
once the notion of an increasing subsequence is properly formulated. Let ip G P m , <fi G P n , 
and denote by i/)\J(f) the concatenation (ipi, . . . , ip m , 0i, . . . , <p n ). Denote by rev(0) the reverse 
of <f>, rev(0) = (0 n , 4> n -i, . . . , 4>\). Given a sequence a from P, an ascent pair 0) for a is 
a pair of subsequences ip of rev(a) and of a such that if ip G P m and G P n then 

1. ip U is weakly increasing with respect to the ordering of P. 

2. For all k G N, at most one (unmarked) k appears in ip. 

3. For all k G N, at most one (marked) k' appears in 0. 

Thus the unmarked symbols are strictly increasing in i/j and the marked symbols are strictly 
increasing in 0. The length of ip U <fi is defined to be m + n — 1. (Note that the length here 
is one less than the length defined in either [HH] or [Sa].) Let a G P n and let L(a) denote 
the length of the longest ascent pair (ip, 0) of a. Then we have 

Theorem (Sagan, Worley). If a is a sequence from P and T is the shifted tableaux of 
shape (Ai, A2, • • • , Afc) resulting from the insertion of a (following the rules of the shifted 
RSK algorithm), then L(a) = Ai. 

Here is an example of a increasing path displayed in the P-matrix A. In this example 
L(A) = 16. 6 



( v 





3' 





2' 


V - 







1' 





1 - 


-> - 


1 

■* 3 





1 


2 





T 
1 








3' 


1 





1' 


T 

2' 


V 











1 




/ 


T 

V 








V 





1' 


T 














I 





T 

V 




- 1' *■ 


- 1' * 


2' ) 



A quick way to compute this length is to apply a modified patience sorting algorithm [AD] 
to the lower row of the associated biword. (We leave this as an exercise to the interested 
reader.) Of course, another way is to apply the shifted RSK algorithm and count the number 
of boxes in the first row. 

6 Note, for example, that the path segment 1' — > 2' in the upper right hand corner contributes a weight 
of two, not three, since the marked elements are strictly increasing. 
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4 A Gessel Identity 



The Gessel identity [Ge] (see also [TW3]) states that the sum 

xev 
\ 1 <h 

equals a certain h x h Toeplitz determinant. The proof begins by expressing the Schur 
functions s\ as determinants (the Jacobi-Trudi identity) and proceeds by recognizing this 
sum of products of determinants as the expansion of a single determinant of the product of 
two (nonsquare) matrices. (This expansion is called the Cauchy-Binet expansion.) 
We are interested in sums of the form 

^Q A (x)P A (y) (4.1) 

\ 1 <h 

where now, as we have seen, the Q\ and P\ are given by pfaffians. What is needed is a 
pfaffian version of the Cauchy-Binet formula. Fortunately, Ishikawa and Wakayama [IW] 
have such a formula. (See also, Stembridge [Ste].) 
Introduce 

I 1 ? = {I = (ii, . . . , i r ) : 1 < ii < ■ ■ ■ < % r < h} 

and denote by Aj the submatrix formed from A by taking those rows and columns indexed 
by J G I^- Then the pfaffian summation formula is 

Theorem (Ishikawa- Wakayama). Let A = (dij) 0<i j <h and B = (%) 0<i j <h be {h+1) x (h+1) 
skew symmetric matrices, h G N. Then 

^ 7 | V(^)p/(5/)+ E E^'V^p/csq/) 

0<r<h j£jh 0<r<h J^zjh 

r even r r odd 7 




where C = (Cy)o<ij<ft is the (h + 1) x (h + 1) antisymmetric matrix 

{1 Jb oj, if i = 0,j> 1, 
fb i0 , ifi >l,j = 0, 
l^hj, ///> I../-'-' I. 

Here \I\ = ^ik and A j, where I = {ii, 12, ■ ■ ■}, stands for Aj where J = {0, i±, 12, . . .}. 

Observe that J/ 1 is the set of partitions with exactly r distinct parts such that the largest 
part is less than or equal to h. For any such partition, r < h. In (4.1) we break the sum 
into two sums — the first sum over distinct partitions with an even number of parts and the 
second sum over distinct partitions with an odd number of parts. Recalling the pfaffian 
representation (2.7) of Q\, we note that if A has an odd number of parts then we extend the 
partition by appending giving us a vector of even length. Thus the sum appearing in the 
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pfaffian summation formula is (up to a reversal of labels) the sum over distinct partitions 
satisfying Ai < h. From (2.7) we see that A^ix) is the (h + 1) x (h + l) antisymmetric matrix 

-q* 

q Qh{x) 

Here q is the h x 1 matrix with elements q r (x) (r = l,2,...,h), Qh{x) is the 
h x h antisymmetric matrix with elements Q( r ,s)( x )> an d Q* denotes the transpose of q. 
(Recall that Q( r ,o) = Qr-) The pfaffian representation for P\ is obtained from the Q\ pfaffian 
representation by inserting the factor 2~ e ^ x \ The matrix is 



B h {y) 





\ q \Qh{y) 



Applying the summation formula then gives for h G N 



(det (J-A h (z)S h (j/))) 



1/2 



The ± factors are accounted for by the reversal of labels; or more simply, because we are 
computing probabilities. 

Introducing the antisymmetric matrix 








) A h (x) 






(o 




(o 




V2 / 




V2 J 



or more explicitly, 



K h (x) 



-^q s (x) r = 0, s > 1, 

-^q r (x) r> l,s = 0, (4.2) 
| Q( r , s )(x) r > 1, s > 1, 



we obtain our Gessel identity: 

1 /2 

Qx{x)Px{y) = [det(I-K h (x)K h (y))j . (4.3) 



Observe that it follows from this and the Cauchy identity (2.3) that 



lim det (I - K h (x)K h (y)) = Z 2 . (4.4) 

h— >oo 
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5 Shifted Schur Measure 

Let V min denote the set of P-matrices of size mxn. For A e V m ^ n we recall that L(A) denotes 
the length of the longest increasing path in A. Let x = (x±, X2, ■ ■ .) and y — (y±, y 2 , ■ ■ .) with 
< Xi < 1 and < yi < 1. We assume the matrix elements a^- are distributed independently 
with a geometric distribution with parameter Xiyj. Specifically, for k > 1 

P (a,, = *) = P (ay = AO = (777^) 

and 

p(«, = o) = ^. 

1 + 2?^ 

We have, of course, 



Let V m ,n,s,t (t £ N m , s G N n ) denote the set of A G P m , n satisfying, for 1 < i < m and 
1 < J < n, 

E a*. = Si and E a*. = tj 

l<j< n l<i<m 

Then for A G P m nst we have 



p (^})= n G 

l<i<m > 



1 - XjJ/jA s f 1 a t 

— — z y = y ■ 

+ x(y.j J Z 



l<j<n 



Since right hand side does not depend upon the A chosen in Vm,n,s,t, the conditional proba- 
bility 



p(l<mE4 = s -E^=^ 



3 

is uniform. Note that this uses both the independence and the geometric distribution of the 
random variables a^-. 

By the shifted RSK correspondence, to each A e V m ,n )S ,t we associate bijectively a pair 
(5, T) of shifted tableaux of the same shape X \= N (N := J2ij a ij)i °f types s and t, 
respectively. The condition L(A) < h becomes Xi < h. Hence 

P m ,n(L<h) = P(£{A)<h\AeV m>n>s>t )P(AeV m , n>s , t ) 

E T^—^^Wm^A 



\ E E Q^) p M 



, 11 ,s,t 

1 

z 



N>0 a|=jv 

X,<h 
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(Here P min denotes probability before a-specialization.) Thus, by (4.3), 

P m ,n (L<h) = ^ (det (J - K h (x)K h (y))y /2 . (5.1) 

The above used the combinatorial definition (2.1) of the Schur Q- function. The reason for 
the occurence of P\(y) (instead of Q\(y)) is that the recording tableaux T has no marked 
elements on the diagonal which accounts for the factor 2~ £ ( x \ (There are exactly 2^ A ) entries 
on the main diagonal in a marked shifted tableaux of shape A and so there are ways to 
mark and unmark the diagonal elements.) 

Observe the consequence that the distribution function P m ,n (L < h) is a symmetric func- 
tion ofx = (xx,...,x m ) and of y = (y x , ...,y n ). 



6 Proof of the Theorem 
6.1 An Operator Formulation 

We begin by deriving an alternative representation for det (J — Kh(x)Kh(y)) in terms of 
Toeplitz and Hankel operators on the Hilbert space £ 2 (Z + ) (Z + := N U 0). These may well 
be of independent interest in the theory of Schur Q-functions. We assume at first only that 
x,y G i 1 together with uniform estimates Xj, yj < c < 1 

To set notation, we let {e^} - >0 denote the canonical basis of £ 2 (Z + ). Since the vector eo 
will occur frequently, we denote eo by e, and it is convenient to set e_i = 0. The backward 
shift operator A is characterized by 

A ej = ej_i 

and its adjoint A* is the forward shift operator. The two satisfy 

AA* = J and A* A = I - e ® e, (6.2) 

where for vectors u and v we denote by u ® v the operator sending a vector / to u (v, f). 
Suppressing temporarily the parameters x and y we define L to be the matrix with entries 

where Q is defined in (2.4). Here the contours can be taken to be concentric circles of different 
radii near the unit circle. Since Q(z) Q(—z) = 1 the residue at £ = —z in the integral defining 
L is zero, so we may freely choose whether the ^-contour lies inside or outside of the ^-contour 
without affecting the value of the integral. ( Equivalent ly, L is symmetric.) 
From (2.5) we see that 

Q(j,k) = Lj-i,k — Lj^-h 
where we set L_i ^ = 0. Note that o = Qj- The matrix elements given in (4.2) are 
then 

~72 L °,k-U j = 0,k^0, 

\{Lj-i ik - L jjk -i) , j,k>0. 
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Introducing the vector q = q(x) = {qo{%), Qi(x), . . .), the operator K can be written 

K = ]-(h*L- LK) + ]-uo(e®q-q®e), (6.4) 

_ — 

where 10 — 1 — y2. 

The operator L is expressible in terms of Toeplitz and Hankel matrices acting on £ 2 (Z + ). 
Recall that T(ip), the Toeplitz matrix with symbol i/j, has j, k entry ipj-k (subscripts denote 
Fourier coefficients here) while the Hankel matrix H(ijj) has j, k entry ipj + f, + x. If we assume 
the contours in (6.3) chosen so that |£| < \z\ and expand (z + Q -1 in powers oi(/z we obtain 

LlJ = f^{-l) k (J-^ f j z-^ k - 2 C k -^ 1 Q(z)Q(OdzdC. 



k=0 



Now make the substitution £ — > — £ to obtain 



yy q(o 



where Q(C) = Q(( The ^-integral gives Q i+ k+i while the (^-integral gives (Q 1 )fc_j. It 
follows that 

L = H(Q)T(Q~ 1 )J, (6.5) 

where J is the diagonal matrix with diagonal entries (— If in the last integrals we make 
the substitutions z — > —2, £ — > — £ we find that also 

L = -J H(Q~ l ) T(Q). (6.6) 

If we reintroduce our parameters x and y, which we now write for notational convenience 
as subscripts, and use the two representations of L we see that 

det (/ + L x L y ) = det (7 - H(Q X ) T(Q~ l ) H{Q~ l ) T(Q y )) 

= det (/ - T(Q y ) H(Q X ) T(Q x l ) H(Q^ . 

Here we have used the general identity det (/ — AB) = det (/ — BA), valid if one of the 
operators is trace class and the other bounded, and the fact that the Hankel operators are 
Hilbert-Schmidt under our assumtions on x and y. Another general fact is 

T(^) H(i> 2 ) + Hty x ) Tfe) = Hty x fo). (6.7) 

In particular, if ipi is a minus function (Fourier coefficients with positive index all vanish), 
then T(ipi) H(ip 2 ) — H(ipiip2)- From this we find that 

T(Q y )H(Q x ) = H(Q x Q y ) and T(Q~ 1 ) H(Qy 1 ) = H^Q" 1 Q y x ), 



14 



so the product of these equals H((p) H(<p 1 ), where 



00) := Qx(z)Q y (z). 



(6.8) 



Since yet another general identity is 



T(^) Tty 2 ) = ^ 2 ) - H^) H(i/j 2 ) 



(6.9) 



we have / — H(<f)) H(<f) *) = T(0) T(0 x ), and we have shown that 



det (I + L x L y ) = detT(4>) T(0 ). 



(6.10) 



If a symbol has geometric mean 1 and is sufficiently well behaved then the strong Szego 
limit theorem says that 



where T h ((j)) = ((f>j-k)j,k=o,...,h-i- In the case of our symbol given by (6.8) we find that 



where Z is as in the right side of (2.3). But there is another formula for E(<p), namely [Wi] 



In the case of Schur functions, the right hand side of the Gessel identity is a Toeplitz 
determinant, and the Cauchy identity for Schur functions emerges as a consequence of the 
Szego limit theorem. In view of the last identity and the connection between the operators 
L and K on £ 2 (Z + ) it is tempting to try to find, using these, an independent derivation 
of (2.3). It will follow from (6.4) that / — K(x) K(y) and / + L x L y differ by a finite rank 
operator. This operator cannot contribute to the determinant but we do not see, a priori, 
why this is so. So such an independent derivation eludes us. 

To continue now, we let Ph be the projection operator onto the subspace of ^ 2 (Z + ) 
spanned by {eo, e±, . . . , e^}. Thus, if K is the operator on £ 2 (Z+) then Kh = PtKPh- Instead 
of working directly with the product K(x) K(y), it will be convenient to write 2x2 matrices 
with operator entries. Thus det (/ — Kh(x) Kh{y)) is the determinant of 





E(4>) — detT(0)T(0~ ), 



and so from (6.10) we have the identity 



det(I + L x L y ) = Z 2 . 



Ph 
P, 



it 



) 



/ K(x) 
K(y) I 



) 



Ph 
P h 



) 
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thought of as acting on P h £ 2 (Z + ) © P h £ 2 (Z + ). To simplify notation, we use Ph to denote 
also 

' P h 
P h 

and set 

K(x 



tffo) 
Thus, 

det (/ - K h (x) K h {y)) = det P h {I + /C)P ft . 

It follows from (4.4) and the infinite-dimensional version of Jacobi's theorem on the 
principal n x n minor of the inverse of a (finite) matrix 7 that this may be written 

det (J - K h {x) K h {y)) = Z 2 det ((/ - P h )(I + K)~\l - P h )) . (6.11) 

Thus our first goal is to compute (I + JC)" 1 . 
Using the easily verified fact 

A*L + LA = q®q-e®e (6.12) 
and (6.4) we see that K = A*L + R~ = -LA + R + where 

R± = 2 ( ±(? ® =F e ® e + w(e ® g - g ® e)) . (6.13) 

Thus 

/ / A*L X + R~ 

I + JC = 

\A*L y + R~ I 
where subscripts have the usual meaning. 



6.2 Calculation of (I + /C) 1 

The fundamental objects which will appear here are / + L x L y and / + L y L x and we begin 
by showing that they are invertible and computing their inverses. Define 

H 1 = H(Q X Q~ 1 ) and H 2 = H(Q y Q~ l ). 

We shall prove the basic identities 

(/ + L x L y y x = I - H X H 2 and (I + L y L x y x = I — H 2 H 1 . (6.14) 

7 The infinite-dimensional result follows by replacing the operator JC by P^ICPn, applying the finite- 
dimensional result and taking the N — > oo limit. We use the fact that JC is trace class, which holds since 
the Hankel operators are trace class. Of course all this requires that the infinite-dimensional operator be 
invertible. This will follow from the limit results we establish, as we shall see at the end of §6.3. 
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We do this with the help of (6.7) and (6.9). Using these and (6.5) we find that 
ifi H 2 L x = H(Q X Q- 1 ) H(Q y Q~ l ) H(Q X ) T{Q x l ) J 

= H(Q X Q- 1 ) [T(Q y ) - T(Q y Q~ l ) T(Q X )] T(Q x l ) J 
= ~H(Q x )T(Q x l ) - H(Q x Q- 1 )T(Q y Q7 x 1 )] J = L x - H x T(Q y Q~ l ) J. 

Thus 

(/ - H x H 2 ) L x = H x T(Q y Q- 1 ) J. (6.15) 
From this and (6.6) we obtain similarly 

(/ - H x H 2 ) L x L y = -Hx T(Q y Q" 1 ) H(Q~ l ) T(Q y ) 

= Hx H(Q y g; 1 ) T{Q~ X ) T(Q y ) = Hx H 2 . 

This establishes the first identity of (6.14), and the second is obtained by interchanging x 
and y. 

Beginning the calculation of (/ + /C)~ x we refer to (6.12) and (6.13) and find that 



I UK 



(J + /C) 



/ + L x L y 

/ + L y L x , 
where we use subscripts as before. Using (6.14) this may be written 



L X AR~ Rt 



LyA I 



L y AR x 



I + L x L y 
I + L V L 



1 + 



(/ - HxH 2 )L x AR y (I - H X H 2 )R+ 
(/ - H 2 Hx)R+ (/ - H 2 Hx)L y AR- 



Hence 



(I + /C) 



-i 



(/ - HxH 2 )L x AR y (/ - HxH 2 )R+ 

(I - H 2 Hx)R+ (I - H 2 Hx)LyAR- x 
I - HxH 2 (I - HxH 2 )L x A 



-i 



x 



(/ - H 2 Hx)L y A I - H 2 Hx 
To compute the entries of the matrix inside the large bracket we show that 

(I-H 1 H 2 )L x Aq y = H l H 2 e and (I — HxH 2 ) q x = 7\ e, 

where we set 

Tx = T{Q x Q~ l ) and T 2 = T(Q y Q x l ). 
For the first we use the fact that Aq = Le, which gives 

(/ - HxH 2 )L x Aq y = (I - H x H 2 ) L x L y e = Hx H 2 e. 



(6.16) 
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To derive the second we use the fact that q x = T(Q X ) e and compute 

H 1 H 2 T(Q X ) = H(Q X Q~ l ) H(Q y Q7 x l ) T{Q X ) = H{Q X Qy 1 ) H(Q y ) 

= T(Q x )-T(Q x Q- 1 )T(Q y ). 
Since T(Q y ) e = e (the matrix is upper-triangular with 0, entry 1) this gives 

Hi H 2 q x = q x - 7i e, 

which is equivalent to the desired identity. Of course the same identities hold if we make the 
interchanges x *-* y and 1 <-> 2. 

With these identities and the fact that AR~ = — |Ag <g> (q + uoe) we find that the matrix 
in large brackets may be written 

j / -HiH 2 e ® (q y + we) T\e <g> (q x - cue) + (I - HiH 2 )e ® (ujq x - e) 

\ T 2 e (g) [q y - Me) + (7 - E 2 H x )e ® (uoq y - e) -E 2 H x e (g) (q x + cue) 
This in turn has the form 

4 

I + y^ j aj®b il 
i=i 

where 

—HiH 2 e \ ^ / —uHiH 2 e + T\e + ue 

= ~ I > fl 2 = - 

-uH 2 H 1 e + T 2 e + ue J A \ -H 2 H x e 

^ / —ujHiH 2 e \ ^ / HiH 2 e — uTie — e 

y H 2 H\e — uT 2 e — e J \ —ujH 2 Hie 

( Qy \ ( 

V / V & 

At this stage we have shown that 

/ 4 v -i / J _ # ljf j 2 (/ - H 1 H 2 )L X A \ 

(I + JC)- 1 ^ [l + J2 a i® b i] • (6.17) 

V i=i / V ( J - HzHjLyA I - H 2 H X ) 

If we have a finite rank operator cij ® 6j, then 

(z + ^Oi^fti) =J-53(S , -%a i ®6 j , (6.18) 











63= 1 






: 




\0) 
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where S is the matrix with entries 

Sij = Sij + (bi, cij)- 

In our case we have to compute 16 inner products, which is not as bad as it might seem 
since there are basic inner products from which the others can be derived. And if we have 
evaluated any inner product then we have evaluated another with the interchanges x <-» y 
and 1 <-> 2. Two basic inner products are trivial: 

(e, e) = 1 and (e, q y ) = 1. 

Two are not evaluable in simpler terms but just notationally. We set 

t = (Tie, e) = (T 2 e, e) and h = (HiH 2 e,e) = (H 2 Hie,e). 

(The equality of the first two inner products follows from the facts that T* = JT 2 J and 
Je = e.) The nontrivial ones are 

(Tie, q y ) = 1 and (H x H 2 e, q y ) = 1 - t. 

For the first, we have (Tie, q y ) = (Tie, T(Q y )e) = (T(Q y )Tie,e), and this is the 0,0 entry of 
T{Qy)T{Q x Q~ 1 ) = T(Q X ). The 0,0 entry equals 1. For the second, we have 

(#i# 2 e, q y ) = 1 - ((/ - H x H 2 )e, q y ) = 1 - (e, (/ - H 2 H X ) % ) = 1 - (e, T 2 e) 

by the second identity of (6.16). 

We can now write down all 16 inner products. For convenience we multiply them by 2: 

2(ai,6i) = 2 (a 2 A) = ut + 1, 2 (a 3 , &i) = -u(l - t), 2 (a 4 , &i) = -t - u - 2, 

2(ai,& 2 ) =wt+l, 2(a 2 ,6 2 ) = -1 +t, 2 (a 3 , 6 2 ) = -t - u - 2, 2 (a 4 , b 2 ) = -u(l - t), 
2 (ai, 63) = — h, 2 (a 2 , 63) = — uoh + t + u, 2 (03, 63) = —uh, 2 (a 4 , 63) = h — out — 1, 
2 (ai, 64) = -w/i + t + lj, 2 (a 2 , 64) = — /i, 2 (a 3 , 64) = /i — cut — 1, 2 (a 4 , 64) = — cj/i. 
Let us see which vectors arise in the end. From (6.17) and (6.18) 

/ / - H X H 2 (I - #i# 2 )L x A \ 4 
(J + /C)- 1 = '>' r (6-19) 
\ (/ - H 2 Hi)L y A I - H 2 H X j i,j=i 

where s^- = (S* -1 )^- and 

/ / - H 2 H X A*L y (I - H X H 2 ) 

V ts?L x {I-H 2 H x ) I-H X H 2 

The quantities that appear in the Oj, other than e which we can ignore since (J — P^)e = 0, 
are H x H 2 e and Tie in the first component and H 2 Hie and T 2 e in the second. Those in the 
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bj are q y and e in the first component and q x and e in the second. For b'- we use (6.16) 
to see that T 2 e, H 2 Hie, h*L y Tie and A*L y (I — HiH 2 )e appear in the first component and 
Tie, HiH 2 e, A*L x T 2 e and A*L X (I — H 2 Hi)e in the second. (The e which appear once again 
drop out in the end.) 

Two new vectors appear here (as well as those obtained by the usual interchanges). We 
claim that 

A*L y T 1 e = T 2 e — te and A*L y (I - H x E 2 )e = H 2 H x e + tT 2 e - (1 - h)e. (6.20) 
For the first, we have 

L y Ti = -JH{Q~ l )T{Qy)T{Q x Qy 1 ) = -JH(Q7 y l )T(Q x ) 

= -JH(Q x Q- 1 )=H(Q y Q- 1 )J, 

so 

A*L y T ie = A*H(Q y Q- l )e = T 2 e - te. 
For the second, we take transposes and interchange x and y in (6.15) to obtain 

L y {\ - H x H 2 )e = JT{Q x Qy l )H{Q y Q x 1 )e 
= -TiQyQ^HiQxQ-^e = H(Q y Q x 1 )T(Q x Q~ l )e. 

So 

A*L y {l - H x H 2 )e = [HizQyQ- 1 ) - e ® T^TiQ^^e 

= E{zQ y Q x l )R{zQ x Q- y x )e - (Tie,T 2 e)e = E 2 E x e + (T 2 e ® T x e)e - (Tie,T 2 e)e. 
The next to last term equals t T 2 e while the last inner product equals 

{JT^JJTxJe.e) = (T(Q :c Q~ 1 )T(QyQ~ 1 )e, e) = ((J — H 1 H 2 )e, e). 

This establishes the second claim. 

It follows from the above that the only vectors that arise in the b'j are T 2 e and H 2 H\e in 
the first component and Tie and H 1 H 2 e in the second. 



6.3 Specialization 

At this point we impose the a-specialization. Thus the first m Xi and the first n jji are 
equal to a and the rest equal to zero. We assume that r = m/n is a constant satisfying 
a 2 < r < a~ 2 and we first determine the asymptotics as n — > oo of the quantities appearing 
in the inner products. We claim 

lim t = and lim h = -. 

n— >oo n— >oo 2 



For the first, we have 



t=Pl)o,„=^/7i±^n— Y-. (6.21) 



27ri y V 1 — ccz / V £ + oc I z 
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If we apply steepest descent we see that the saddle points are to satisfy 

r 1 

0, 



1 — a 2 z 2 z 2 — a 2 
and so are given by 

/ l_ Q 2 r 

z = ±1 \ r = ±1/9 - 

V t — cr 

These are purely imaginary under our assumption on r. The steepest descent curve passes 
through these points and closes at a and —a -1 . (See Fig. 1.) The integral is 0(n~ 1 ^ 2 ). For 
the second, we have 

ii \ m / \ n /ii > \ 7i / > 
1 + az\ ( z — a\ ( 1 + a.Q \ ( Q — a 



z + aj \1 — a( J \( + a 

-V fff^Y ( z ^Y (i±^Y (^Y z-^c^^ (622) 

2m) J J \l-az) \z + aj \1 - a( J \(+a) z( - l' V ' ; 

where the contours are such that \z(\ > 1. Setting i — j — and making the substitution 
C — > C 1 gives 

1 \ [l + az\ (z — a\ (Q + a\ /1 — a C\ dzd(. 



2j0,0 



27ri/ J J \\ — az J \z + a J \C — a J \1 + a£ J z(z — Q 



where now on the contours (both still described counterclockwise) \z\ > |£|. If we ignore the 
z — C in the denominator we have two integrals to each of which we apply steepest descent. 
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The saddle points are ±i/3 for both. The new z contour is as before but the new ( contour 
closes at —a and a -1 . We first deform the original £ contour to this, always remaining inside 
the original z contour. Then we deform the z contour to its steepest descent curve. In the 
process we pass through the points of the ( contour from —i/3 to i(3 in the right half-plane. 
The z residues at these points are l/( and so the deformations lead to the double integral 
over the steepest descent contours, which is C^n" 1 / 2 ), plus (2-ni)~ l J\„ d(/( = |. This 
establishes the second limit. 

When uj = 1 - V2 we find that det S has the limit (5 - 3\/2)/8 ^ as t -> 0, 
h — > 1/2. 8 Hence the entries of S^ 1 are all bounded. (The invertibility of S for large n 
implies in turn the invertibility of / + JC and hence the validity of the determinant identities 
we have been using.) 



6.4 Scaling 

If we write P for / — P^ then we see from (6.19) that P{I + /C)~ X P equals the identity 
operator / minus 

PE X E 2 P \ / P(I - E x E 2 )L x kP 

PE 2 E 1 P J \P{I- E 2 E 1 )L V AP 

4 

+ s ij Pa i ®Pb' :j . (6.23) 

Eventually we will set i = h + ra 1//3 x, j = h + n l ^y where h = cn + n^^s with c to be 
determined. The operators PE\ and E\P will give rise to integrals like 



+ olz \ I z — a 



z - cn -n^x dz 



1 — az J \z + a r 
(with a different x) and PE 2 and E 2 P will give rise to integrals like 



1 + az \ I z — a 



1 — az J \z + a y 

If we make the substitution z — >• z~ x in the latter we get an integral like 



z — a \ l + az 



z + a\ (l-az\ +n i/ Sx 



m 

dz. 



If we set 

% f I + az\ m f z - a\ n 



cn 



8 Computations verify that when r < a 2 the limit of ft. is and t = (— l) m + o(l), and that the limit of 
det 5 is (9-4V2)/2 as 77i — > oo through even values and — 1/2 as m— > oo through odd values. For the limits 
when r > a~ 2 we replace m by n. The proofs should be similar to what we have already done, except that 
the saddle points will now be real. 



22 



our integrals become 



4>(z) z~ nl/3x dz and J ij(z)- 1 z nl/3x dz. 

If we think of the factors ^(z)^ as the dominant ones and apply steepest descent, there will 
in general be two saddle points for the two integrals, and the product of the critical values 
will be exponentially small or large. If c is chosen so the critical points coincide 9 then the 
product of the critical values will be 1 and the product of the operators will have nontrivial 
scaling. 

To determine c, let a(z) = n~ l logip(z), so that 

.. . Iolt la c 
<J (z) = - + — j - -• 

1 — a z z z z z — cr z 

If we eliminate c from cr'(z) = u"{z) = we obtain 

r(l + a 2 z 2 ) a 2 + z 2 



a 2 z 2 ) 2 (z 2 — a 



2\2 



0. (6.24) 



The function on the left is strictly increasing from — oo to +oo on the interval (a, a -1 ). It 
follows that there is a unique point z in this interval where the function vanishes. This will 
be our saddle point and we set 

c=2az ( T + 1 ) >0. (6.25) 
\1 — a Zq Zq — a z J 

From the behavior of cr'(z) for large negative z and near —a -1 , a and we see that 
er' has a zero in (— oo, — a" 1 ) and a zero in (— a,0). Since it has a double zero at zq this 
accounts for all four of its finite zeros. Since <j'(z) tends to +oo at the endpoints of (a, a~ l ) 
it follows that it is positive everywhere there except at Zq, and this implies that a"'(z Q ) > 0. 
(Well, this only shows that a"'(z ) > 0. We find in the last section an explicit expression for 
a"'(zo) in terms of zo, from which it is clear that it is positive.) 

The two steepest descent curves, which we call T + for the first integral and T~ for the 
second, together form a single contour. The first emanates from z at angles ±7r/3 with 
branches going to oo in two directions. The second emanates from z$ at angles ±27r/3 and 
closes at z = 0. See Figure 2. 

It will be convenient to replace £ 2 ([h,oo)) by £ 2 (Z, + ), and to do that we change our 
meaning of the operator P. A P appearing on the left is to be interpreted as A h and a P 
appearing on the right is to be interpreted as A* h . So, for example, the i th component of 
PTie is (Tie)h+i and the i,j entry of PH\Pi2 is (H 1 H 2 )h+i,h+j- With these reinterpretations 
of P the operators in (6.23) all act on £ 2 (Z + ) and the determinant has not changed. (Recall 
that h = cn + n 1 / 3 ,?.) 

Let D be the diagonal matrix with i th diagonal element equal to ip(z )~ 1 ZQ 1/3s+l , and 
multiply (6.23) by ( J on the left and by ( ^ ^ j on the right. This will not 



3 This c will be the ci(a, t) of the Theorem. 
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affect the determinant. (The reason is that all the operators in (6.23) have i, j entry 0{r %+ ^) 
for fixed n, where r can be any number larger than a, and a < zq < a' 1 .) We shall show 
that after these multiplications the first operator scales to the the direct sum of two Airy 
operators, 10 the second operator has trace norm o(l) and the vectors in the sum will all have 
norm o(l). 

6.4.1 The first operator in (6.23) 

The upper left corner of the first operator becomes, aside from the identity operator, DPHiH 2 PD 
We write this as 

(DPHiDo) {D^H 2 PD~ l ), 

where D is the diagonal matrix with ith diagonal element equal to z z , and we scale each 
factor. We have 

77,1/ 3 rj 



-1 



27ri J T v ' z 2 
The main fact will be the following. Define 

V 1 — az \z + a 



${z) z-^s-'-i %. (6.26) 



10 If a matrix acting on £ 2 (Z + ) has entries M(i,j) and if the kernel n 1/ ' 3 M([n 1 / 3 x], [rt 1 / 3 ?/]) acting on 
L 2 (0, oo), which is unitarily equivalent to the matrix operator, converges in trace norm to a limiting kernel 
then we say that the matrix scales in trace norm to the limiting kernel. The Frcdholm determinant of M(i,j) 
then converges to the Fredholm determinant of the limiting kernel. This is what will happen here, with the 
limiting kernel being the direct sum of two Airy kernels. 
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and set 

the contour being the unit circle. Then 

V^o)" 1 zf'^n 1 ' 3 \I n {x)\ < e~ 5x (6.27) 
valid for some 5 and all n if x is bounded from below, and 

lim ^(^o)" 1 zf 3x n 1 ' 3 I n (x) = z^gAi(gx) (6.28) 

n— >oo 

pointwise, with g the constant given by (6.31). (The limit in (6.28) will be uniform for x in 
a bounded set.) 

We first show that (6.27) holds if n 1 ^ 3 x > r\n for some r] > 0. For this we set 7 = c+n~ 2 ^ 3 x 
so that 7 — c > rj, write our main integrand as i/)(z, 7) and do steepest descent. With 

a(z, 7) = n~ l log i/j(z, 7) = a(z) — (7 — c) logz, (6.29) 

our saddle points z^j (there will be two of them when 7 > c) are solutions of a'(z^,j) = 0. 
Differentiating this with respect to 7 gives 



07 ^ 



7 

Since a" (2^, 7) 7^ (since c is the only value of 7 for which there is a double saddle point) 
we find that if z^ are chosen so that z* > z~ for 7 near c then 2;+ increases always as 7 
increases and z~ decreases, and a1(z+,j) > and a"(z~,j) < 0. In particular 2+ is the 
saddle point we take for our steepest descent, and z+ > z when 7 > c since z+ = z . For 
the critical value we have to see how a(z^,j) behaves as a function of 7. From (6.29) and 
the fact 7) = we obtain 



g?7 

and so 



- log 



T"[^7>7) + 7 tog*)] = log% (6.30) 

07 z 31 



7 



Since is an increasing function of 7 this shows that <r(z+,7) + 7logzo is a decreasing 
concave function of 7. It follows that for some 5 > 

cx(z+,7) +7log2;o < a(z ) + clog z - 5 (7 - c) 

for 7 > c + r/. Thus for these 7 

^(z )- 1 ^ (7 " C) ^«,7)<e-^- c) . 
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Since \ip(z, 7)] achieves its maximum on the steepest descent contour at z 7 , and since the 
contour is bounded away from zero (this follows from the fact that z 7 — > a as 7 — > 00), we 
see that in this case I n (x) is at most a constant times i/j(z+, 7), where 7 = c + n~ 2 / 3 :r. Hence 

^o)" 1 zf 3x \I n (x)\=0(e- 5nl/3x ), 

when n 1 / 3 ^ > r]n. This is an even better estimate than (6.27). 

Since we have shown that (6.27) holds if n 1//3 x > r)n, where 77 can be as small as we 
please, we may assume n 1//3 x = o(n) when x > 0. 

Write 

'.<*> = si /*M^""-£ 

and use the steepest descent curve T + . It emanates from z at angles ±7r/3. Clearly it is 
bounded away from 0. Choose e small and let 

T« = {z£ r + : \z — zo I > and T (2) = {z G T+ : |z - z \ < e} 

with corresponding In\x) and In\x). We shall show that both of these satisfy the uniform 
estimate (6.27) and 

lim ^(zor 1 n 1/3 l£\x) = and lim ^(z^ 1 zf' x n 1/3 1^ (x) = z^gki{gx). 

n— >oo n— >oo 

Consider In\x) first. Since $ta(z) is strictly decreasing as we move away from z on T 
we know that |^>(z)| < ip(zo)e~ 5n on for some 5 > 0. The assertions follow from this 
since we are in the case n 1//3 x = o(n). (This also holds also for x < since x is bounded 
below.) 

Making the variable change z — > z§ (1 + £) we can write, since 2: = z e^ 1+ °^ near 
z = z 0) 

= ^) z- 1/3 --L / e n6^3 (1+o(0) _ nl /3 1BC(1+o(0) ( a + o(0) ^ 

2tti 7|€|< e /^ 

where b = a"'(zo)/6. The path of integration here is the portion of the contour satisfying 
the indicated inequality. It consists of two little arcs emanating from £ = tangent to the 
line segments making angles ±7r/3 with the positive axis. If e is small enough and we replace 
the integral by the line segments themselves we introduce an error of then form 0(e _<5n ) with 
a different 5, since r^^'^x = o(n). With the variable change £ — > n _1 / 3 £ we obtain 

^z )-'zf 3 *n^l£\x) 
= — [ e K 3 * 3 (i+o(n-^0)-*«i+o(n-v^))^-i + Oin" 1 /^)) d£ + 0(e- 5n ), 
where now the integration is taken over line segments of length of the order n 1 ^ 3 . 
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On the path of integration we have 5i(£ 3 ) < — <5|£| 3 and 9?(£) > 5\£\ for some 5 > and 
so for the above we have an estimate of the form 



POO 

/ e - s( - t3 - xt) dt + 0(e 
Jo 



5n\ 



Since n 3> x this gives the required uniform bound. The limit of the integral, with its factor 
l/27ri, equals Zq 1 g Ai(gx), where 

^(3^)- 1/3 -» I (^) I/3 - (6.3D 

That the limit is as stated follows by taking the limit under the integral sign, which is 
justified by dominated convergence. 

To obtain the scaling of the matrix DPH\Dq we need only observe that by (6.26) 

(DPH.Do)^^ = ^o)- 1 4 1/3 ^ 1/3 ^ 1/3 *1 n^Us + n-^dn^x] + [n^y])). 

It follows from (6.27) and (6.28) that this kernel on (0, oo) converges in Hilbert-Schmidt 
norm to z$ 1 g Ai(g(s + x + y)). 
To scale D Q ~ l H 2 PD~ 1 we write 

— n 1 / 3 ^— i— j /•/-I, \ n / \ m 

1/3 s-i-jruzr\ z o l + az\ (z-a\ „_ m _ n i/3, 



If we make the substitution z z this becomes 



— n 1 / 3 s— i— j 
^ / ^(^)- 1 ^ 1/3s+ ^'d 2 . 



2vri 

This is completely analogous to (6.26). To use the analogous argument we mention only that 
we use (6.30) with the minus signs to see that both —a(z~j) —7 log z and its derivative are 
decreasing functions of 7. The steepest descent curve now is T~. We need not go through 
the details again. We find that 

n 1/3 (D^PHiD'^^i/s^fai/Sy] 

converges in Hilbert-Schmidt norm to z$g Ai(g f (s + x + y)). Hence 

n^iDPH^PD- 1 )^^^ 

converges in trace norm to g K^ iry (g(s + x), g(x + y)) on L 2 (0, 00). 

By taking transposes we see that D~ 1 PH2H\PD~ 1 has the same scaling limit, which 
takes care of the lower right corner of the first operator in (6.23). 
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6.4.2 The second operator in (6.23) 

Next, we have to look at 

D P(I — H 1 H 2 )L X AP D and D~ x P(I - H 2 H 1 )L y KP IT 1 . 
We find, using (6.15), 

z 2nV*s + i +j{{I _ HlH2)LxA)h+ . h+ . 

\ I + az i i z — a\ ( 1 + aQ\ I Q — a 



27ri y J J \1 — az J \z + aj \1 — a( J \(^ + a 

„ i/o . . ~-h-i-l fh+j-l 

xzl n/s+ ^- — ^ d^dC(-l) h+J . 



After the substitution £ — > — C -1 this becomes 



±Y(f, {z)m (±v""-'f<v" n -> 



The integrals here are initially taken over circles close to the unit circle, with \z\ > \(\. We 
first deform the £ contour to T + , while always having z + ( nonzero. Then if we deform the 
z contour we pass through a pole at C, = — z for every z ET + . The residue at the pole equals 
a constant times z ~ 2h ~ l ~^~ 2 ^ and integrating this over r + gives zero. So both integrals may 
be taken over T + . Since now the denominator does not vanish at z = ( = z Q the same sort 
of argument we already gave shows that this operator equals a constant times ip(z ) 2 n -1 / 3 
times an operator which scales to the trace class operator gAi(g(s + x)) <g> gAi(g(s + y)). 
In particular its trace norm is O(^(^ ) 2 n" 1/3 ). This shows that D P(I - H 1 H 2 )L X AP D has 
trace norm 0(n -1 / 3 ) and a similar argument applies to D~ l P(I — H 2 H\)L y AP D~ l . 

6.4.3 The last operator in (6.23) 

Finally we consider the vectors Pcij and Pb 1 - and look at their constituents PT^e, PT 2 e, 
H\H 2 e and H 2 H\e. We shall show that 



-l/6i 



||I>.PTie|| = 0(n" 1/6 ) and ID^PT^W = 0(n 

||Mi# 2 e|| = 0(n~ 2/3 ) and \\D- 1 H 2 H 1 e\\ = 0(n _2/3 ) 
For the first, we have 



(T ie ) h+i = ^-J i>{z) z - 1/3 ^ dz = I n (s + n- x '\i - 1)), 

and from (6.27) and (6.28) we deduce now that the function 

n^iDPT.e)^ 
converges in L 2 (0, oo). In particular its norm is 0(1). But then 

WUDPT^Up = n 1 ^ WiDPT.e)^,^ = 0( n -^). 
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Similarly \\D- l PT 2 e\\ = 0(n~ l l Q ). 
For DHiH 2 e, we have from (6.22) 



1 \ ffl 1 + az \ f z — a\ (l + a(\ (C-a\ _ h ^ x „i/3 s+! ,_ 1 d^d( 



2ni J J J \1 — az J \z + a J \1 — aQ J \C, + a J z( — 1 

and with the substitution ( — > £ _1 this becomes 

Si)'//*" (6,2) 

As before the integrals here are initially taken over circles close to the unit circle, with 
\z\ > |C|- Now we want to deform the z contour to r + and the ( contour to its steepest 
descent contour C, a curve passing through the saddle points ±i/3 _1 and closing at —a and 
a- 1 . 

To do this we show first that, except for zq, all points of T + satisfy \z\ > z . This will 
follow if we can show that on the circle z = Zq e l6 the absolute minimum of 



log 



az \ I z — a 



az J \ z + a 



(6.33) 



occurs at 9 = 0. (For then ^(z)! would be larger than ijj(zo) everywhere on the circle except 
for z = Zq, so no other point on the circle could be on T + . Locally r + is outside the circle 
and so it would have to be everywhere outside.) Using 

d . d 

d6 dz 

we find that the derivative with respect to 9 of (6.33) equals —2a times the imaginary part 
of 

t 1 _ (r - a 2 )z + (1 - tol 2 )z~ x 

z^ 1 — a 2 z z — a 2 z~ 1 (z~ l — a 2 z) (z — a 2 z~ 1 ) 
This vanishes exactly when the imaginary part of 

((r - a 2 )z + (1 - a 2 )z^) (z' 1 - a 2 z) (z - a 2 ^ 1 ) 

does. This is a trigonometric polynomial in 9 of degree 3. It is an odd function of 9 and so 
is of the form sin 9 times a polynomial of degree two in cos 9. Since it has at least a double 
zero at 9 = (by the choice of c and zq) the polynomial must have a factor cos 6* — 1. Since 
it is an odd function of z it must also have a double zero at 6 — tt, so there must also be a 
factor cos# + 1. Thus it must be equal to a constant times sin# (cos 2 9 — 1). In particular 
there can be no other zeros. Thus (6.33), which we know has a local minimum on the circle 
at z — Zq, must have its absolute minimum there (and its absolute maximum at z — —z ). 
Thus, as claimed, all points of T + except for z satisfy \z\ > z 

In particular, all points of T + satisfy \z\ > zq. Since zq > a we can first take the integrals 
in (6.32) over the circles \z\ = zq and |£| = a + e with e small and positive. Then we can 
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deform the z contour to T + without crossing the circle |£| = a + e. Next we want to deform 
the ( contour to C. This curve closes on the right at a -1 and so, since Zq < a -1 , it intersects 
T + at two points z' and z", say. (In principle there could be finitely many other points; 
the following argument could be easily modified in this case.) Hence upon deforming the £ 
contour to C we pass through a pole for those z on the arc of T + passing through z = zq 



with end-points z' and z" . For each z on this arc the residue at £ 
and then integrating with respect to z gives 



z equals z 



1 n^-^s+i 



2, 







7i/3, 



(h 



h+i 



h+i 



We claim that -^(zo) -1 (which is the factor contained in D) times this vector is exponentially 
small, i.e., 0(e~ Sn ) for some 5 > 0. 

Because all points of T + except for z satisfy \z\ > z the vectors {(zq/z") 1 } and {(zq/z') 1 } 



belong to t 
same for z" 



(Z + ). So we need only show that ip(z ) 



is exponentially small (and the 



In fact, on the part of C in the right half-plane 

z + a\ (I 



a 



az 



1 + az 



c < (1 - 5) l^(z') | 1/n for some 5 > 0. Since \ip(z)\ < \ip(z )\ on T+ this 
1—5) ^(zo)! 1 /™ and it follows that iP(zq)" 1 \z'\~ h is exponentially small. 



is at most 1 and is strictly less than 1 outside a neighborhood of the critical points ±if3. 
This shows that \z' 

shows that \z'\~ c < [i — 0) i^^ojr''" ana it roiiows tnat ip{Zo) " \z 

We can now say that, with error ip(zo) 2 times an exponentially small quantity, the square 
of the norm of the vector (6.32) equals a quadruple integral in which every term in the 
integrand except z~ h ~ % ~ x z % Q has an analogous term with variables z', and the z~ h ~ l ~ l z l 
term becomes (zz')~ h / (zz' — Zq). For the z and z' integrals we integrate over T + and for the 



C, and £' integrals we integrate over C . (We use again here the fact that 



\Z I > Zq, SO 



we can sum under the integral signs.) The £, £' integrals contribute 0{n ) while the 
integrals contribute O(ip(z ) 2 n" 1 / 3 ). Thus WDPH^eW 2 = Oin'^ 3 ). 11 



For H2H\e we interchange m and n and make the variable change z 



_1 but not the 



variable change C, — > C • Thus 



,1/3, 



a( 



(H 2 H ie ) h+ 

c 



a 



,1/3 



s+i 



7.V3, 



dzdC 



— a( J \( + a t 

Here originally we must have \C\ > \z\ on the contours and we want to deform them so 
that the ( contour becomes T~ and the ( contour becomes what we again call C. Now in 
the deformation, we pass through a pole in the ( integration for those z on an arc of T~ 
passing through z = z with end-points which we again call z' and z". The residue equals 
and integration with respect to z gives 



~h+i „ 



7.V3, 



{h + i + iy 



z 



11 h+i+l 



Z 



I h+i+l 



n This was under the basic assumption a 2 < t < oT 2 . Otherwise the ( and (' integrals are only O(l), 
with the result that \\DH1H2eW is only 0(n -1 / 6 ) and the same will hold for \\D~ 1 H2Hie\\ . These are still 
good enough since o(l) is all that is needed. 
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This is completely analogous to what went before. Now \z'\, \z"\ < z Q and ijj(zo) \ z '\ h is 
exponentially small. We continue as with H 1 H 2 e and find that \\D~~ 1 PH2Hie\\ 2 = 0(n~ 4 / 3 ). 

If we go back to the forms of the Oj and b'- described earlier we see that the vectors which 
arise after multipliying by the diagonal matrices D and D~ Y are exactly the four whose 
norms we just estimated. 



6.5 Recapitulation 

We have shown that the matrix (6.23) acting on £ 2 (Z + ) scales in trace norm to the kernel 
g K Airy (g(s + x), g(x + y)) 

g K Airy (g(s + x), g(x + y)) 

acting on L 2 (0, oo). It follows that its Fredholm determinant converges to F{gsf. In view 
of (5.1) and (6.11) this establishes that for fixed s 

lim P CT (L <cn + n 1/3 s) = F 2 (gs), 

n—*oo 

where c is determined by (6.24) and (6.25) and g by (6.31). This gives the statement of the 
Theorem, where the constants ci(a,r) and 02(01, t) of the Introduction are, respectively, c 
and g~ x . 



6.6 Computation of a"'(z ) 

Think of c and z as functions of r, which they are. We have 



a"(z) 



Aaz 



a 2 r 



1 



(l-a 2 z 2 ) 2 (z 2 -a 2 ) 2 



.2 ' 



Differentiating the identity = o-"(z ) with respect to r and using the above give 



= a"'(z )z' + 



4a 3 zq 



+ 



(1 — a 2 z 2 ) 2 z { 



2 • 



where z' and d denote dz /dr and dc/dr, respectively. 

From (6.24) and (6.25) we find that c is given in terms of z by the relation 



We compute that 



dc 
dz 



Aa{l-a 4 )zl 
°~ (l + a 2 z 2 )(z 2 -a 2 ) 2 - 

a (1 - a 4 ) zl [zl + 3a 2 + 3a 2 ^ + a 4 z 2 ) 



(l + a 2 z 2 ) 2 (z 2 -a 2 ) 3 
From (6.24) r is given in terms of Zq by 

(l-a 2 z 2 ) 2 (a 2 + z 2 ) 



T 



(l + a 2 z 2 ) {z 2 -a 2 ) 2 ' 
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(6.34) 



(6.35) 



We compute that 



dr_ 

dz 



2(1- a 4 ) z (1 - a 2 zl) (a 4 z 2 + 3a 2 z 4 + 3a 2 + zg) 
(z 2 -a 2 ) 3 (l + a 2 z 2 ) 2 



(6.36) 



We first solve (6.34) for a'"(z ) in terms of z' and d (and z ). Then we use (6.35) and 
(6.36) and the relations z' = (dr/dzo)" 1 and d = (dc/dz ) z' . We find that 



This is positive since < a < 1 and a < z < a 1 . 

7 Poisson Limit of the Shifted Schur Measure 

For Schur measure there are two interesting limiting cases: the exponential limit and the 
Poisson limit. The exponential limit of shifted Schur measure is supported on standard 
shifted tableaux; and hence, it is expressible in terms of fg (recall (2.2)). The Poisson limit 
of shifted Schur measure yields a natural interpretation of the maximizing rule as a symmetry 
condition of the process. 12 Namely, if one sets 



and takes n — > oo, the percolation-type model described in §3 becomes the following: Con- 
sider two Poisson processes of rate t 2 both in the (same) square [0, 1] x [0, 1]. Hence one can 
imagine two types of points, marked and unmarked, in the square. Now with probability one 
no two points, whether marked or unmarked, have the same x or y coordinates; and hence, 
the strictly increasing conditions (2) and (3) of §3 for the rule of the maximizing path are not 
necessary. Therefore, we do not need to distinguish the unmarked and marked points. Since 
the union of two Poisson processes has the rate 2t 2 , the reulting process is as follows: In the 
square [0, 1] x [0, 1] select Poisson points of rate 2t 2 . Then take the longest path starting 
from the lower right corner (1,0) that follows an up/left path, turns in direction once and 
only once, and follows an up/right path ending at the upper right corner (1, 1). 

It is clear that this length is also equal to the following symmetric version. Take a re- 
alization of the Poisson process. Take the mirror image of the points about the left side. 
Adjoin the mirror image on the left and the original points on the right. Hence the resulting 
rectangle has sides of lengths 2 and 1, and there are twice as many points of the orginal con- 
figuration which are symmetric about the center vertical line. The (usual) longest up/right 
path from the left lower corner to the right upper corner is precisely the longest maximizing 
path from the lower right corner to the upper right corner in the above description. 

The (formal) limit m = n — > oo with a = t/n (t fixed) in the main theorem of §1 is 



a'"(z )=4 



a (1 - a 4 ) [(1 + a 4 ) z 2 + 3a 2 (1 + z 4 )] 
(1 - a 4 z 4 ) (z 2 - a 2 f 



t 

m — n, a = — 
n 




(7.1) 



12 Thc following remarks are due to the referee. 
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The consequence is that the vertically symmetric Poisson process has the same fluctuation 
as the usual Poisson process with no symmetry condition. Also the scaling in the above 
result is consistent with this intuition. There are Poisson points of rate 4t 2 (double of 2t 2 ) 
in the square of sides 2 and 1. For such the case the limit (7.1) is also valid for the case of 
no symmetry condition. 
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